Papers with co-attention mechanism
Two-Headed Monster and Crossed Co-Attention Networks (2020.aacl-srw)
Copied to clipboard
| Challenge: | a new co-attentional neural structure is proposed for machine translation tasks . a higher-level and more abstract paradigm generalized from CCNs is proposed . |
| Approach: | They propose a paradigm that consists of two symmetric encoder modules and one decoder module connected with co-attention. |
| Outcome: | The proposed model outperforms the current Transformer model on translation tasks but the epoch time increases by circa 75%. |
Reinforced Cross-modal Alignment for Radiology Report Generation (2022.findings-acl)
Copied to clipboard
| Challenge: | Medical images are widely used in clinical decision-making, where writing radiology reports can be enhanced by automatic solutions to alleviate physicians’ workload. |
| Approach: | They propose an approach with reinforcement learning over a cross-modal memory to better align visual and textual features for radiology report generation. |
| Outcome: | The proposed approach improves cross-modal alignment on two English radiology report datasets and human evaluation confirms the results. |
Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for sarcasm detection ignore the incongruity character in sarcasm, which is often manifested between modalities or within modalités. |
| Approach: | They propose to capture inter-modality incongruity in a text-based model by using a self-attention mechanism and a co-attention model to model the contradiction within the text. |
| Outcome: | The proposed model achieves state-of-the-art on a public multi-modal sarcasm detection dataset. |
Leveraging Gloss Knowledge in Neural Word Sense Disambiguation by Hierarchical Co-Attention (D18-1)
Copied to clipboard
| Challenge: | Existing models for Word Sense Disambiguation use labeled data, but lack gloss knowledge. |
| Approach: | They propose a co-attention mechanism to generate co-dependent representations for context and gloss . they propose to incorporate gloss knowledge into neural networks for Word Sense Disambiguation . |
| Outcome: | The proposed model achieves state-of-the-art results on standard English all-words WSD datasets. |
A Joint Learning Framework for Restaurant Survival Prediction and Explanation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning have various models that research reviews and interactions for different kinds of tasks, such as predicting restaurant survival. |
| Approach: | They propose a joint learning framework for explainable restaurant survival prediction based on multi-modal data of user-restaurant interactions and users’ textual reviews. |
| Outcome: | The proposed framework improves on two datasets showing that it can model restaurant interactions and users’ textual reviews. |
On the Automatic Generation of Medical Imaging Reports (P18-1)
Copied to clipboard
| Challenge: | a complete medical imaging report contains multiple heterogeneous forms of information, including findings and tags . abnormal regions in medical images are difficult to identify and the reports are typically long, containing multiple sentences. |
| Approach: | They propose a multi-task learning framework which predicts tags and generates paragraphs for abnormal regions in medical images. |
| Outcome: | The proposed framework can generate long paragraphs on two publicly available datasets. |
Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn effective alignment between vision and language features are insufficient in practice due to complicated multi-step reasoning. |
| Approach: | They propose an iterative alignment network which iterates inter- and intra-modal features within multiple steps for more accurate grounding. |
| Outcome: | The proposed model performs better than the state-of-the-arts on three challenging benchmarks. |